Memory workflow management in edge devices

ABSTRACT

Techniques discussed herein relate to providing in-memory workflow management at an edge device (e.g., a computing device distinct from and operating remotely with respect to a data center). The edge device can operate as a computing node in a computing cluster of edge devices and implement a hosting environment (e.g., a distributed data plane). A work request can be obtained by an in-memory workflow manager of the edge device. The work request may include an intended state of a data plane resource (e.g., a computing cluster, a virtual machine, etc.). The in-memory workflow manager can determine the work request has not commenced and initialize an in-memory execution thread to execute orchestration tasks to configure a data plane of the computing cluster according to the intended state. Current state data corresponding to the configured data plane may be provided to the user device and eventually displayed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This non-provisional application claims priority to U.S. Pat.Application No. 63/173,244, filed on Apr. 9, 2021, entitled “CloudComputing Edge Computing Device (Rover),” the disclosure of which isherein incorporated by reference in its entirety for all purposes.

BACKGROUND

In cloud computing, processing and storage is generally performed by oneor more service providers implemented at a centralized location. Datacan be received from customers at the centralized location, processedthere, and then the processed (or other) data can be transmitted back tocustomers. However, having a centralized location for cloudinfrastructure components may not be ideal in various scenarios. Forexample, when there are hundreds or thousands of Internet of Things(IoT) devices transmitting data to the central servers, and especiallywhen those IoT devices are not geographically close to the cloudinfrastructure computing devices, conventional centralized systems arenot ideal. These IoT devices may be considered on the “edge,” as in theyare not close to the central servers.

Additionally, there may be other instances when the centralized locationfor cloud components is less than ideal. For example, if the data iscollected (e.g., by IoT devices) in a disconnected region or a locationwith no Internet connectivity (e.g., remote locations). Currentcentralized cloud computing environments may not meet time sensitivityrequirements when streaming data due to the inherent latency of theirwide-area network connections. Remotely generated data may need to beprocessed more quickly (e.g., to detect anomalies) than conventionalcentralized cloud computing systems allow. Thus, there are challengeswith managing a traditional cloud computing environment that relies oncentralized components. For example, a centralized workflow manager maybe suboptimal for managing workflows at geographically remote devices.

BRIEF SUMMARY

Techniques are provided (e.g., a method, a system, non-transitorycomputer-readable medium storing code or instructions executable by oneor more processors) for providing in-memory workflow management at acloud infrastructure edge computing device (e.g., a computing deviceconfigured to deliver computing and storage at remote locations separatefrom the centralized data center and lacking a public/private networkconnection). Various embodiments are described herein, includingmethods, systems, non-transitory computer-readable storage media storingprograms, code, or instructions executable by one or more processors,and the like.

One embodiment is directed to a method for providing in-memory workflowmanagement at an edge computing device. The method may compriseimplementing a plurality of respective hosting environments within acomputing cluster comprising a plurality of edge computing devices. Insome embodiments, the plurality of edge computing devices implement adistributed control plane for managing respective data planes of theplurality of edge computing devices. The method may further compriseobtaining, by an in-memory workflow manager of the distributed controlplane, a work request comprising a request identifier and intended statedata corresponding to a respective data plane of a particularcloud-computing edge device of the plurality of edge computing devices.The method may further comprise determining, using the requestidentifier, that the work request has yet to be started. The method mayfurther comprise initializing, by the in-memory workflow manager, anin-memory execution thread to execute one or more orchestration tasksassociated with configuring the respective data plane according to theintended state data. The method may further comprise obtaining currentstate data indicating a current state of the respective data plane ofthe particular cloud-computing edge device. The method may furthercomprise providing an indication of the current state of the respectivedata plane.

In some embodiments, the work request is initiated by from a user deviceconfigured to communicate with one cloud-computing edge device of theplurality of edge computing devices. The in-memory workflow manager andthe in-memory execution thread, in some embodiments, execute within aDocker container on the particular cloud-computing edge device.

In some embodiments, the one or more orchestration tasks associated withconfiguring the respective data plane according to the intended statedata are executed in response to: 1) obtaining, from a distributed datastore accessible to the plurality of edge computing devices,previously-received actual state data associated with the respectivedata plane of the particular cloud-computing edge device, 2) comparingthe previously-received actual state data and the intended state data,and 3) determining that there is a difference between thepreviously-received actual state data and the intended state dataobtained from the distributed data store.

In some embodiments, identifying that the work request has yet to bestarted comprises determining that a record associated with the requestidentifier is not yet stored in a distributed data store (e.g., in aportion of the distributed data store configured to store requestidentifiers of requests that have already been started) accessible tothe plurality of edge computing devices.

The plurality of edge computing devices may be communicatively coupledwith one another via a substrate network that is different from a publicnetwork. In some embodiments, the in-memory workflow manager, executingon the particular cloud-computing edge device, identifies the one ormore orchestration tasks associated with configuring the respective dataplane according to the intended state data.

In some embodiments, an edge computing device is disclosed. The edgecomputing device may operate as part of a computing cluster of aplurality of edge computing devices. In some embodiments, the edgecomputing device comprises one or more processors and one or more(non-transitory) memories configured with computer-executableinstructions that, when executed by the one or more processors, causethe edge computing device to perform operations. These operations maycomprise implementing a hosting environment within the computingcluster. In some embodiments, the plurality of edge computing devicesimplement a distributed control plane for managing respective dataplanes of the plurality of edge computing devices. The operations mayfurther comprise obtaining, by an in-memory workflow manager of the edgecomputing device, a work request comprising a request identifier andintended state data corresponding to a respective data plane of aparticular cloud-computing edge device of the plurality of edgecomputing devices. The operations may further comprise determining,using the request identifier, that the work request has yet to bestarted. The operations may further comprise initializing, by thein-memory workflow manager, an in-memory execution thread to execute oneor more orchestration tasks associated with configuring the respectivedata plane according to the intended state data. The operations mayfurther comprise obtaining current state data indicating a current stateof the respective data plane of the particular cloud-computing edgedevice. The operations may further comprise providing an indication ofthe current state of the respective data plane.

Some embodiments disclose a non-transitory computer-readable storagemedium comprising computer-executable instructions that, when executedwith one or more processors of an edge computing device (e.g., an edgecomputing device operating as part of a computing cluster of edgecomputing devices, cause the edge computing device to perform the methoddisclosed above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example high-level architecture for acloud infrastructure edge computing device, according to at least oneembodiment.

FIG. 2 is a block diagram of an example architecture for connecting auser computing device to a cloud infrastructure edge computing device,according to at least one embodiment.

FIG. 3 is a block diagram of an example enclosure for a cloudinfrastructure edge computing device, according to at least oneembodiment.

FIG. 4 illustrates an exploded view of the cloud infrastructure edgecomputing device described herein, in accordance with at least oneembodiment.

FIG. 5 is a block diagram of an example computer architecture of a cloudinfrastructure edge computing device, according to at least oneembodiment.

FIG. 6 is a block diagram depicting a distributed computing cluster thatincludes one or more edge computing devices, according to at least oneembodiment.

FIG. 7 is a block diagram depicting a flow for executing a workflow byone or more components of a cloud infrastructure edge computing device,according to at least one embodiment.

FIG. 8 is a block diagram depicting a hosting environment provided by anedge computing device, according to at least one embodiment.

FIG. 9 is a block diagram depicting a number of hosting environmentsprovided by a computing cluster including two edge computing devices,according to at least one embodiment.

FIG. 10 illustrates an example flow for performing one or moreoperations in connection with a work request, in accordance with atleast one embodiment.

FIG. 11 is a block diagram illustrating an example method for providingin-memory workflow management at an edge computing device, in accordancewith at least one embodiment.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

Introduction

In some examples, a cloud-integrated edge service (e.g., implemented inan edge computing device) may be integral in addressing the desire torun time-sensitive cloud infrastructure application outside of acentralized data center (e.g., a datacenter of a cloud infrastructureservice provider). Such an edge computing device may deliver computingand storage at the edge and/or in disconnected locations (e.g., remotelocations separate from the centralized data center and lacking apublic/private network connection (e.g., an Internet connection, a VPNconnection, a dedicated connection, etc.) to enable low-latencyprocessing at or near the point of data generation and ingestion. Insome instances, a fleet of portable (which may be ruggedized forprotection) server nodes (e.g., a fleet of edge devices) may beconfigured to physically bring the cloud infrastructure service toremote locations where cloud technology has been consideredtechnologically infeasible or too cost prohibitive to implement.

To a customer (e.g., a user), the edge computing device can act as anextension of their cloud infrastructure: including virtual machines(VMs), containers, functions and data files, block volumes or objectstore services can also be delivered from the cloud infrastructuretenancy (e.g., a tenancy of the centralized cloud computing environment)with little to no modifications, and the customer experience may remainunchanged from that of the centralized cloud computing experience.Additionally, the edge computing device may be configured to implementboth a control plane and a data plane that are part of a cloudinfrastructure service provider. The data plane can be configured tomanage data storage, migration, processing, etc., while the control plancan be configured for controlling the various services and architecturecomponents of the computing device. Once the edge computing device isproperly connected to a customer computing device (e.g., via a localarea network (LAN)), the customer may be able to utilize the IaaSservice (or at least a subset of it) using the same SDK and API usedwith the centralized cloud service.

The edge computing device can be delivered to a customer in apre-configured form, such that the only action that might be required ofthe customer is to connect the nodes to a network (e.g., a local/onpremise network that is accessible by a user computing device), powerthem up, and/or log in. The device can be pre-configured in various waysbased on customer preference/request, or it can be in one of variousconfigurations (e.g., storage-centric, compute-centric, etc.). The nodeor cluster of nodes can be portable and is intended to be mobile - whenmoved and set up again (or used while in motion), the deploymentcontinues to run from where it turned off (or continuously). The edgecomputing device can also monitor for wide area network (WAN) connectionavailability (e.g., the Internet or the like), and can synchronizecustomer and management data with the cloud once connected to a WAN.

Some potential use cases for the edge computing device include: storageand processing, compute and input/output (I/O) intensive applications,machine learning, remote computing, low latency database and analytics,and data collection and migration. More specifically, the edge devicecan be used for storage and processing of large volumes of images,video, audio, and IoT sensor data generated in environments where WANconnection is latent or unavailable (e.g., in remote areas, an oiloff-shore platform, or the like). Once this data is pre-processed,filtered, compressed, and/or secured it may be transported ortransferred to the cloud service provider, where it can be furtherprocessed by the centralized server (e.g., traditional cloud serviceprovider). The device can also be used for compute and I/O intensiveapplications, where low latency is paramount, such as tacticalreconnaissance or 5G communications. The device can also be used formachine learning, with models trained in the cloud and running indisconnected locations to improve efficiency, intelligence, and/orproductivity in manufacturing, document management, transportation, oiland gas mining, and/or telecommunications. It can also be used forremote computing requiring elevated security and airtight containment ofdata. Additionally, the device can be used for low latency database andanalytics workloads, with more applications optimized over time.Further, the device can also be used for data collection and migrationof large sets of object and database management system (DBMS) data intoa cloud service provider, e.g., at faster speeds and lower cost than aWAN transfer.

The edge device can natively support distributed cloud paradigms, wherecomplex, multi-stage compute workflows can be separated into individualcomponents, which in turn can be deployed to the infrastructure of theedge device, on premise, and/or in the cloud. An example of suchdistributed workflow is represented in the following scenario. Massiveamounts of data can be collected by an edge computing node deployed onan airplane (e.g., a military jet) in a reconnaissance operation with noInternet access (e.g., a disconnected edge computing device), where thisdata is be pre-processed in near real time by a machine learning modelpreviously trained by the cloud service provider that provided the edgedevice. Even the first pass of processing the data with the models candetect significant anomalies and can alert personnel immediately - forexample, a bridge may be destroyed and therefore the troops should bererouted. When the airplane lands, the edge computing device can bephysically connected to a network (e.g., an edge station potentiallydeployed at the airstrip). The pre-processed, filtered, smaller datasetcan be loaded for final processing to a cluster of edge computing devicenodes at the edge station. The original edge computing device can bereleased and can be loaded on another (or the same) airplane, forexample to support the next mission. When processing at the edge stationis complete, a 3D map update can be issued for immediate use. Changesets can then be uploaded by the edge station cluster to a datacenterand can be used to build future models providing intelligent tacticalforecasts to the reconnaissance operation, or the like.

It should be appreciated that the following techniques may be employedin a variety of contexts such as telecommunications, oil and gas,healthcare, hospitality, agriculture, transportation, and logistics, andthe like.

Embodiments described herein address these and other problems,individually and collectively. Specifically, embodiments of the presentdisclosure provide for a cloud infrastructure edge computing device.

Edge Device Architecture

An edge computing device (sometimes referred to as “a cloud-computingedge device,” a “cloud infrastructure edge computing device,” or an“edge device,” for brevity), extends a user’s centralized cloudcomputing tenancy by physically putting customer infrastructure andplatform services where data is generated - on the edge, on premise, orcompletely disconnected. Each deployment is created to address specificcustomer needs by provisioning VM instance images and data from thecustomer’s centralized cloud tenancy. These workloads remain fullyfunctional offline as the edge device adapts to the connection state,operates in harsh environmental conditions, and is ready to sync withthe cloud whenever the connection is reestablished.

FIG. 1 is a block diagram of an example high-level architecture for acloud infrastructure edge computing device (e.g., edge device 100),according to at least one embodiment. An overview of the software andhardware component of the edge device 100 is provided below.

In some examples, the edge device 100 may include containerizationengine 102 (e.g., Docker, Kubernetes, etc.) configured to implement oneor more containers (e.g., corresponding to container(s) 104A, 104B,104C, to 104N, collectively referred to as “container(s) 104”). Acontainerization engine (e.g., the containerization engine 102) may becontainer-orchestration system for automating computer applicationdeployment, scaling, and management. In some embodiments, thecontainerization engine may be configured to provide OS-levelvirtualization to deliver software in packages called containers. Thesecontainers can be isolated from one another and utilize respectivesoftware, libraries, and configuration files, and can communicate witheach other through well-defined channels. In some embodiments,service(s) 104 may include any suitable number of services (e.g., one ormore). These services may implement at least some portion of centralizedcloud capabilities. Each service may be stand-alone or operate as adistributed cluster. The edge device 100 may further include ahypervisor 106 configured to implement one or more virtual machines(e.g., virtual machines 108A, 108B, 108C, to 108N, collectively referredto as “virtual machine(s) 108” or “VMs 108”).

In some examples, the edge device 100 includes storage 110 (e.g., objectand/or block storage for storing local data). The edge device 100includes operating system (OS) 112. In some embodiments, the OS 112 maybe optimized for executing on an edge device and/or specific toexecution on an edge device. OS 112 may be configured to manage thehardware of edge device 100 and supports a data plane of the servicesrunning on the edge device 100. The OS 112 may be configured to supporta specific deployment type (e.g., a single edge device deployment, or aspecific edge device cluster configuration). The OS 112 may beconfigured to secure the edge device by disallowing or otherwiseblocking direct access by customers.

In some embodiments, the edge device 100 may include hardware such asany suitable number of central processing units (CPUs) and/or storagedrives. For example, the edge device 100 depicted in FIG. 1 may haveone, two, or more CPUs, with various numbers of cores per processingunit, and it may include any number of storage drives (e.g., 6.4terabyte (TB) drives, or the like). As a non-limiting example, the edgedevice 100 may include block and/or object storage of any suitable size.The edge device 100 may include any suitable number of centralprocessing units (CPUs), graphics processing units (GPUs), random accessmemory (RAM) of any suitable size, one or more ports (e.g., QSFP28,RJ45, dual ports, etc.), tamper-evident seals, or any suitablecombination of the above components.

In some examples, the basic system functionality/services can beaccessed via RESTful APIs have a custom load of software based on Linux.The virtual machine(s) 108 may individually be a Kernel-based VirtualMachines (KVM) (e.g., a virtual machine managed by a virtualizationmodule in the Linux kernel that allows the kernel to function as ahypervisor) and/or a hardware-based Virtual Machine (e.g., a virtualmachine managed by a virtualizer, such as Quick EMUlator (QEMU), thatcan perform hardware virtualization to enable virtual machines toemulate of number of hardware architectures). Although storage 110 isrepresented as a separate component from the service(s) 104 and VM(s)108, it can run as a container (e.g., container 104A) or in a VM (e.g.,VM 108A). In some examples, it may be favorable to implement the storage110 (e.g., object storage, block storage, etc.) as a container.

FIG. 2 depicts an example architecture 200 for connecting the edgedevice described herein (e.g., edge device 100 from FIG. 1 ) to acomputing device 202 (e.g., a user computing device). The computingdevice 202 can be any type of computing device including, but notlimited to, a laptop computer, a desktop computer, or the like. The edgedevice 204 (an example of the edge device 100 of FIG. 1 ) may includecontainerization engine 206 (an example of the containerization engine102 of FIG. 1 ), hypervisor 208 (an example of the hypervisor 106 of 1),and storage 210 (an example of the storage 110 of 1).

Additionally, as mentioned briefly above, the edge device 100 mayinclude an API proxy 212 for managing the RESTful API calls receivedfrom the computing device 202. The API calls may enter the edge device204 via network interface card (NIC) 214 that is internal to the edgedevice 204. The NIC 214 may be used to connect the edge device 204 tothe computing device 202 via a local area network (e.g., the LAN 216).The API calls received by the NIC 214 may be transmitted to an exposedendpoint that may implement a Web server (e.g., endpoint 218). The webserver can transmit the requests to the API proxy 212, which can routethe requests to the appropriate service (e.g., containerization engine206, hypervisor 208, and/or storage 210). The exposed endpoint/webserver may also be configured to implement the lightweight console thatis for use by the customer (e.g., the user interface displayed on thecomputing device 202).

The lightweight console can run within a web browser (e.g., MozillaFirefox, or the like) on a laptop computer, desktop computer, or othernetwork-accessible device (e.g., connected to the local area network(LAN 216)) that is network-connected to the edge device 204 (e.g., via arouter, cable, etc.). The edge device 204 can expose the endpoint 218for the console connection, and the web server can transmit data to theweb browser of the computing device 202 over the LAN 216.

FIG. 3 illustrates an example physical enclosure 300 of the edge devicedescribed herein (e.g., edge device 100 from FIG. 1 ). Various differentform factors, shapes, colors, etc., can be employed to build a box(e.g., ruggedized) that can house the edge computing device. Thephysical enclosure can include handle 302, as shown, and may includetamper evident elements, so that if anyone breaks the enclosure open, itwill be evident. In this way, the service provider that provides theedge computing device can ensure that the device is not modified. Insome examples, the physical enclosure 300 may not be possible to open.However, in some cases, it might be possible, but it would requireextreme measures.

FIG. 4 illustrates an exploded view of the cloud infrastructure edgecomputing device described herein (e.g., edge device 400, an example ofthe edge device 100 of FIG. 1 ), in accordance with at least oneembodiment. The various components described with respect to FIGS. 1 and2 can be communicatively attached to one or more motherboards and/orinterface cards within the edge device 400. The illustratedconfiguration of components is but just one implementation. The specificlocations of components shown is not intended to be limiting, and asnoted, any configuration that is capable of implementing thefunctionality described herein is acceptable. Once the components areinstalled, the entire box can be closed, sealed, and locked withtamper-evident components.

The edge device 400 is a single enclosure. The enclosure may be designedto house any suitable number of serially attached SCSI (SAS) solid-statedrives (SSDs) and all other components (e.g., CPU, memory, GPU, etc.)within the enclosure. The system may include one or more (e.g., 12 Gb)SAS connections to each drive in a fully contained sheet metal enclosuredesigned to fit within a standard 19″ rack resting on an Lbracket/shelf, on a table top or upright next to a desk with the use ofa floor stand.

The system may include a tamper evident enclosure, front security plugscovering screws holding a front bezel in place with rear securityinterlock features. In some embodiments, the system may include a dualsocket motherboard and any suitable amount of DRAM. In some embodiments,the system may include any suitable number (e.g., 2, 3, etc.) SATA SSDs,storage controllers, embedded network connections, one or more ports(e.g., dual ports, serial ports, etc.), one or more fans as part of acooling system, or any suitable combination of the above.

As a non-limiting example, the edge device 400 may be made up of anexternal extruded aluminum case secured in the front with a vented bezeland rear panel only exposing I/O connections required for data transferand management. Mounting can be designed to mount the any suitablemotherboard, fans, and power supply.

FIG. 5 is a block diagram of an example computer architecture of a cloudinfrastructure edge computing device (e.g., edge device 500, an exampleof the edge devices 100 and 204, of FIGS. 1 and 2 , respectively),according to at least one embodiment. The edge device 500 can be thoughtof as a cloud-integrated service that extends some or all ofconventional cloud capabilities to locations that may not be accessibleby or have access to cloud data centers. This can be achieved viaportable ruggedized server nodes that provide cloud-like functionalityin locations with no WAN connectivity. This allows customers to shiftselect cloud workloads to remote locations and enable intensive dataprocessing operations close to the data ingestion points at the edge oftheir cloud infrastructure.

The edge device 500 may include any suitable number of services (e.g.,service(s) 502). Each service may run as a container (e.g., a Dockercontainer) locally on the edge device 500. The service(s) 502 may becommunicatively connected via a substrate network 504 such that thecommunications between services are encrypted (e.g., in accordance witha security protocol such as MACsec). Each container may be assigned asubstrate IP address (e.g., a static address) with which traffic can beaddressed. In some embodiments, a security protocol (e.g., MACsec) isconfigured at provisioning time (e.g., before the edge device 500 isshipped to the user). The edge device’s system software (includingservice(s) 502) may execute in the secure environments protected by bootsecurity software (e.g., Trenchboot Secure Launch). Users may berestricted from accessing the secure environment and/or the substratenetwork 504. To minimize the amount of resources used by these services,the service code may be compiled and saved to disk to decrease RAM spaceas well as decrease the CPU load on the edge device 500.

Some example services included in service(s) 502 may include a UIconsole service, an identity control plane (CP) service, an identitydata plane (DP) service, a compute application programming interface(API) service, a compute worker thread service, a virtual network (VN)API service, a block storage API service, a function-as-a-serviceservice, an events service, an object storage management service (e.g.,implementing a storage platform such as Ceph Storage or the like), acompute DP service (e.g., an example of hypervisor 208 of FIG. 2 ), a VNDP service, a block storage management service, a function-as-a-serviceAPI service, a function-as-a-service load balancing (LB) service, afunction-as-a-service process thread service, a distributed data storemanagement service (e.g., etcd3), a dynamic host configuration protocolservice, a domain name system service, a network time protocol (NTP)service, to name a few. Some example functionality provided by theseservices is discussed below.

By way of example, compute DP service may be configured (e.g.,preconfigured and provisioned onto the edge device 500) to isolate theVM(s) 508 on the same hypervisor host. The compute DP service canutilize any suitable container engine (e.g., Docker container,MicroContainer, or the like) to isolate the VM(s) 508 on the samehypervisor host from each other. The compute DP service may utilize anysuitable hypervisor (e.g., Quick EMUlator (QEMU), Kernel-based VirtualMachine (KVM), etc.) to provide virtual hardware emulation for VM(s)508. In some embodiments, VNIC(s) 506 are attached to subnets of anysuitable number of virtual networks (e.g., private virtual network(s)(PVN(s))) 505 and are assigned private Internet Protocol (IP) addresses.One VM may have multiple VNICs from different VCNs and differentsubnets. The maximum number of VNICs can be limited by predefinedthresholds (e.g., configuration data referred to as “VM shape” thatdefines VNICs per VM count, VNIC shape, etc.). In some embodiments, thepredefined thresholds are applied to each of the VM(s) 508. The subnetsutilized by the VNIC(s) 506 may be isolated by VLANs. In someembodiments, some or all of the VNIC(s) 506 may be assigned publicand/or private IP addresses. A public IP address is an address in thenetwork 520, while a private IP address refers to an IP address of thePVN(s) 505.

In some embodiments, the edge device 500 implements various networkingfunctionality via a number of services such as a network addresstranslation (NAT) service, a dynamic host configuration protocol (DHCP)service, a domain name system (DNS) service, a network time protocol(NTP) service, a metadata service, and a public API service). Themetadata service may provide initialization data and other metadata toall VM(s) 508. In some embodiments, DHCP service assigns private IPaddresses to each of the VNIC(s) 506, each of the VM(s) 508 having oneor more VNICS. DNS service may provide domain name resolution to VM(s)508 on the edge device 500. NTP may provide time synchronization toVM(s) 508. In some embodiments, a public IP service executing as part ofservice(s) 502 may enable a VM to access a public API without assigningthe VM a public IP and without configuring a service gateway.

In some embodiments, at least one of the VM(s) 508 may implement block(or object) storage. In some embodiments, the hypervisor associated witha virtual machine may include a library that enables the hypervisor touse a distributed data storage platform (e.g., Ceph). The library mayutilize a protocol associated with that storage platform (e.g., RADOSBlock Device (RBD) to facilitate storage of block-based data. Thedistributed data storage platform may be implemented over multiplevirtual machines. In some embodiments, the distributed data storageplatform supports making snapshots and copying block volumes. VM imagesand VM block volumes can be Ceph block devices. In some embodiments, theVM(s) implementing the distributed data storage platform will usesystem-reserved resources (e.g., eight CPU cores, or any subset of thetotal number of CPUs available on the edge device 500). For example inorder to provision a boot volume, a block device image may be copied toa boot volume of the block device. The distributed data storage platformmay use block devices that include multiple nodes for redundancy. Ifsome node fails then the block device can continue to operate. In someembodiments, the distributed data storage platform (e.g., Ceph or thelike), automatically recovers the block device data in case of a fewnode failures. Block storage may be utilized to store images for anysuitable deployable resource. By way of example, an image may beutilized for launching VMs. In some embodiments, the image maycorrespond to a particular VM shape (e.g., a compute heavy VM, a GPUoptimized VM, a storage VM, and the like).

Compute API service may support the following operations: 1) VM launchand terminate, 2) VM stop, start, reboot, 3) List VMs and/or getinformation on a specific VM, 4) obtain VM console history API, 5)obtain a VM snapshot, 6) attach/detach block volumes, and the like. Insome embodiments, Compute API service can be used to call other services(e.g., compute DP service, identity DP service for authentication andauthorization, etc.).

Some of the functionality of other services will be discussed inconnection with FIG. 7 . In general, although each service may not bediscussed in detail herein, the general functionality provided by theservice(s) 502 may include the functionality of cloud services providedby a remote cloud service provider. In some embodiments, the edge device500 may be associated with a predefined region and/or realm such thatsome of the service(s) 502 may operate as if they were operating in acloud computing environment, despite the fact they are operating on oneor more local device(s) (one or more edge devices) as a single instanceor as part of a distributed service that may have no or intermittentpublic network access to a cloud computing environment associated withthe customer. A “region” refers to a geographic location at which aservice center resides. A “realm” refers to a logical collection ofregions. Realms may be isolated from each other and do not share data.

In some embodiments, the edge device 500 may provide any suitable numberof virtual networks (e.g., PVN(s) 505) using compute, memory, andnetworking resources (e.g., virtual network interface card(s) (VNIC(s)506)). A virtual network is a logical network that runs on top of aphysical substrate network. Using the service(s) 502, one or morecustomer resources or workloads, such as virtual machines (e.g., virtualmachine(s) (VM(s)) 508, executing a compute instance) can be deployed onthese private virtual networks. Any suitable combination of VM(s) 508can execute functionality (e.g., a compute instance, storage, etc.)which is individually accessible through a virtual NIC (e.g., one of thevirtual NIC(s) 506). Each VM that is part of a PVN is associated with aVNIC that enables the VM (e.g., a compute instance) to become a memberof a subnet of the PVN. The VNIC associated with a VM facilitates thecommunication of packets or frames to and from the VM. A VNIC can beassociated with a VM when the VM is created. PVN(s) 505 can take on manyforms, including peer-to-peer networks, IP networks, and others. In someembodiments, substrate network traffic of the service(s) 502 may beencrypted and/or isolated (e.g., by virtue of different PVNs or subnets)from network traffic of one or more the VM(s) 508 executing on the edgedevice 500.

The edge device 500 thus provides infrastructure and a set ofcomplementary services that enable customers to build and run a widerange of applications (e.g., compute instances), services, and/orstorage in a highly available, physically local, and virtual hostedenvironment. The customer does not manage or control the underlyingphysical resources provided by the edge device 500 but has control overexpanding or reducing virtual machines (e.g., compute instances, virtualNICs, block or object storage, etc.), deploying applications to thosevirtual machines, and the like. All workloads on the edge device 500 maybe split into different CPU sets (e.g., VM and non-VM). One set (e.g.,non-VM such as workloads performed by the service(s) 502) may utilize asubset of CPU cores (e.g., 8) of the edge device 500, while the otherset (e.g., VM workloads performed by the VM(s) 508) may utilize adifferent subset of CPU cores.

The edge device 500 may be communicatively connected to a user device(e.g., the computing device 202 of FIG. 2 ) via one or more networkinterfaces (e.g., NIC2 and/or NIC 4) and network 520 to interact and/ormanage the VM(s) 508. In certain embodiments, a lightweight console canbe provided at the user device via a web-based user interface that canbe used to access and manage the edge device 500. In someimplementations, the console is a web-based application (e.g., one ofthe service(s) 502) provided by the edge device 500.

FIG. 5 depicts a single edge device. However, it should be appreciatedthat more than one edge device may be utilized as a distributedcomputing cluster.

FIG. 6 is a block diagram depicting a distributed computing cluster 600that includes one or more edge computing devices (e.g., edge device 602and 604, each an example of the edge device 500 of FIG. 5 ), accordingto at least one embodiment.

Each edge device of the distributed computing cluster 600 may beconnected via substrate network 606 (an example of the substrate network504 of FIG. 5 . In some embodiments, the edge devices of the distributedcomputing cluster 600 (sometimes referred to as “edge computing nodes”or “edge nodes”) may be connected by the substrate network 606 using oneor more switches (e.g., switch 608 and/or 610). In some embodiments,NIC1 and NIC5 may include a particular connector (e.g., RJ45 connector)while NIC3 and NIC8 may include the same or a different connector (e.g.,a QSFP28 100 GbE connector). In some embodiments, only one edge deviceof the distributed computing cluster 600 is connected to a customernetwork such as network(s) 620 (an example of the network 520 of FIG. 5). Thus, not only may traffic between services of an edge device beencrypted and isolated from other traffic of a given edge device, buttraffic between distributed services operating across multiple edgedevices may also be encrypted and isolated from other traffic of thecomputing cluster. In some embodiments, each edge device ispreconfigured as a particular node in the distributed computing cluster600. In other embodiments, the user can configured the number andtopology of the edge devices of the distributed computing cluster 600.

FIG. 7 is a block diagram depicting a flow 700 for executing a workflowby one or more components of a cloud infrastructure edge computingdevice, according to at least one embodiment. Components that executethe flow 700 may include API service 702, database (DB) 704, workerservice 706, hypervisor service 708, PVN CP service, Block storage CPservice 714, although more or fewer services may be included. In someembodiments, each of the services of FIG. 7 are an example of a serviceof the service(s) 502 of FIG. 5 . In some embodiments, at least some ofthe functionality discussed in connection with the services of FIG. 7may be combined in any suitable combination and provided as a singleservice or instances of the same service. By way of example, in someembodiments, the functionality of services 702-708 may be provided by asingle service (e.g., compute CP service discussed above in connectionwith FIG. 5 ). In some embodiments, the functionality provided by theservices 702-708 may be provided by a single edge device (e.g., edgedevice 500 of FIG. 5 ) or by two or more edge devices (e.g., by edgedevice 602 and edge device 604 of FIG. 6 ).

In some embodiments, the API service 702 may be configured to acceptwork requests that include intended state data that describes anintended state of a set of data plane resources (e.g., VM(s) 508 of FIG.5 ). As a non-limiting example, user 720 may utilize a user device(e.g., the user device *202 of FIG. ^(∗) 2 ) to access a user interfacewith which he can make various selections indicating a desire to launcha VM. The user input may be received by the API service 702 (an exampleof the compute CP service of FIG. 5 ) which may generate a work request(WR) (e.g., WR 722) and utilize a predefined Launch VM API to store thework request in a distributed database (e.g., DB 704). In someembodiments, the DB 704 may be a computing cluster, which is configuredto use etcd3 as an immediately consistent, highly-available,transactional, distributed database. Generally, a work request indicatesa desire and information needed to create and/or modify data planeresources such as VM(s) 508. In some embodiments, the work requestincludes state information indicating a desired state for the data planeresource. In some embodiments, the DB 704 may be accessible to allservices operating on any edge device (and by services operating on anysuitable edge device of an edge device cluster such as distributedcomputing cluster 600).

Worker service 706 (e.g., an example of the compute CP service of FIG. 5) may be configured to execute one or more worker processes (e.g., oneor more computing threads, such as computing thread 710). Some of theseworker processes may be configured by the worker service 706 at anysuitable time to execute a continuous and/or ongoing predefinedworkflow. By way of example, the worker service 706 may configure one ormore worker threads (e.g., including computing thread 710) to monitorthe DB 704 for new work requests (e.g., WR 722). The computing threadmay be configured to determine if a work request WR 722 is already beingattended to. In some embodiments, this entails checking a predefinedstorage bucket within DB 704 for a unique identifier associated with WR722. If the unique ID included within WR 722 does not appear in thebucket (or the WR is otherwise indicated as having not been picked upfor processing), the computing thread 710 (e.g., a nanny thread) mayinitialize a workflow thread (e.g., another instance of a computingthread 710) which may then be configured by the computing thread 710 toexecute a workflow corresponding to launching a VM corresponding to theWR 722.

The initialized workflow thread may be communicatively coupled (e.g.,via the substrate network 504 of FIG. 5 ) to a workflow service (notdepicted). The workflow service may be configured to identify, from oneor more predefined workflows, a predefined workflow that corresponds tolaunching a VM, and therefore, to the WR 722. These predefined workflowsidentify one or more steps/operations to be taken, and a sequence tothose steps, in order to achieve a predefined goal (e.g., launching avirtual machine, stopping/starting a virtual machine, terminating avirtual machine, creating a block volume, removing a block volume,etc.). The workflow thread may launch the VM workflow and oversee itsexecution by various other entities. In some embodiments, the workflowthread may pass any suitable portion of the intended state data of theDP resource to any suitable combination of services.

As a non-limiting example, as part of the workflow for launching avirtual machine (e.g., a VM to be hosted by hypervisor service 708), oneor more APIs can be called for creating and attaching the VNIC.Similarly, a number of APIs may be provided for creating and/orattaching a block storage volume API. In some embodiments, the workflowthread may perform any suitable call to one or more APIs to invoke thefunctionality of PVN CP Service 712, which in turn may be configured tocreate and attach a VNIC. The workflow thread may then call blockstorage CP service 714 which may then execute any suitable operations tocreate and attach a block storage volume. The worker thread overseeingthe workflow may ensure a designated order (e.g., create the VNIC firstbefore creating the block volume). This worker thread may be configuredto catch any errors and/or exceptions from one or more services it hasinvoked. If no exceptions/errors are encountered, the worker threadoverseeing the workflow can provide any suitable data to the hypervisorservice 708 (via the substrate network), which in turn, executefunctionality for creating the VM requested. The hypervisor service 708may provide actual state data for the newly launched VM. In someembodiments, the worker thread overseeing the workflow can store theactual state data in the DB 704 for later reference (e.g., when amonitor may determine whether the actual state data matches therequested state data indicating no changes needed or when the actualstate data fails to match the requested state data, indicating a changeof the data plane resources is needed).

In some embodiments, the workflow thread may be communicatively coupledto a cluster manager (not depicted). Cluster manager may be configuredto manage any suitable number of computing clusters. In someembodiments, the cluster manager may be configured to manage anysuitable type of computing cluster (e.g., a Kubernetes cluster, a set ofcomputing nodes used to execute containerized applications, etc.). Theworkflow thread may be configured to execute any suitable operations tocause the cluster manager to execute any suitable orchestrationoperation on the DP resource(s) (e.g., a VM) in accordance with theinstructions identified to bring the DP resource(s) in line with theintended state data. In some embodiments, a monitoring entity (e.g., theworkflow thread, a thread launched by the workflow thread) may becommunicatively coupled to DP resource(s) 116 and configured to monitorthe health of DP resource(s). In some embodiments, the monitoring entitymay be configured to store any suitable health data in the DB 704.

The specific operations and services discussed in connection with FIG. 7is illustrative in nature and is not intended to limit the scope of thisdisclosure. The particular operations performed and services utilizedmay vary depending on the particular workflow associated with therequested operations.

FIG. 8 is a block diagram depicting a cloud-computing environment 800provided by an edge computing device (an example of the edge devices ofFIGS. 1-6 ), according to at least one embodiment. Cloud-computingenvironment 800 may include control plane 802 and data plane 804.

In some embodiments, the control plane 802 may be responsible foraccepting work requests that include intended state data that describesan intended state of a set of one or more data plane resources. Forexample, a work request may be received by control plane applicationprogramming interface (API) 806. The work request can be initiated byuser 808 via a user device 810 interfacing with an edge computing device(not depicted) at which control plane 802 and data plane 804 operate. Insome embodiments, control plane API 806 (e.g., an example of the APIservice 702 of FIG. 7 ) may be one of the service(s) 812 (an example ofthe service(s) 502 of FIG. 5 ). Control plane API 806 may be configuredto receive any suitable number of work requests corresponding to one ormore data resources (e.g., a virtual machine, a cluster of virtualmachines, etc.) from the user device 810.

A work request may include a request identifier and intended state data.The request identifier may uniquely identify the work request such thatthe work request can be distinguishable from other work requests. By wayof example, the request identifier for a particular work request can bean alphanumeric string of characters of any suitable length that isunique to that work request and with which that work request can beidentified. Intended state data may include any suitable number ofparameters. These parameters may define attributes of the data planeresource requested including, but not limited to, an identifier for theresource, an availability domain, a shape corresponding to the node, anumber of processing units of the resource, an amount of random accessmemory (RAM) of the resource, an amount of disk memory, a role (e.g., adata node, a master node, etc.), a status (e.g., healthy), or the like.In some embodiments, the control plane API 806 may be configured tostore all received work requests in a data store (e.g., a distributeddata store) configured to store such information (e.g., control plane(CP) data store 814).

In some embodiments, CP data store 814 may be configured to store workrequests and/or an intended state data corresponding to an intendedstate of the data plane 804. In some embodiments, the CP data store 814may be configured to store a mapping of one or more data planeidentifiers (DPIDs) of DP resource(s) 816 with intended state dataand/or current state data. Intended state data refers to data thatspecifies one or more aspects of a DP resource which has been requestedand to which the DP resource is intended to be modified. Current statedata (sometimes referred to as “actual state data”) corresponds to oneor more parameters that identify one or more current aspects of a DPresource as currently operating.

The control plane 802 may include a control plane (CP) monitoringcomponent 818. The CP monitoring component 818 may be configured toperiodically (e.g., according to a predetermined frequency, schedule,etc.) determine whether the intended state data received by the controlplane API 806 and stored in the CP data store 814 (e.g., from apreviously received work request) matches current state data stored fora corresponding DP resource (if that DP resource currently exists). CPmonitoring component 818 may be communicatively coupled to non-computeservice(s) 812 (e.g., via substrate network 504 of FIG. 5 ) which mayinclude any suitable number of cloud computing services configured tomanage billing, identity, authorization, and the like. In someembodiments, CP monitoring component 818, in-memory workflow manager820, and CP workers 822 are provided by a common service (e.g., workerservice 706 of FIG. 7 ). Worker service 706 may be one of the service(s)502 of FIG. 5 . Non-compute service(s) 812 may be the remaining set ofservices of service(s) 502, excluding control plane API 806 and aservice (e.g., worker service 706) that implements CP monitoringcomponent 818, in-memory workflow manager 820, and CP workers 822. Insome embodiments, CP monitoring component 818 is at least part of one ofthe service(s) 812. In some embodiments, CP monitoring component 818 maybe communicatively coupled to in-memory workflow manager 820 and may beconfigured to invoke the functionality provided by the in-memoryworkflow manager 820. By way of example, if the CP monitoring component818 determines that the current state data of a DP resource is not inline with (e.g., does not match) the intended state data stored in CPdata store 814, CP monitoring component 818 may invoke the functionalityof in-memory workflow manager 820 to rectify the discrepancy.

In some embodiments, in-memory workflow manager 820 may be configured toidentify one or more predefined workflows which individually identifyoperations to perform to configure DP resource(s) 816 in accordance withcorresponding intended state data. In some embodiments, the in-memoryworkflow manager 820 may be configured to initiate one or more workersof control plane (CP) worker(s) 822 and forward the workflowinstructions and/or intended state data to a given CP worker to performthe operations related to configuring the corresponding DP resource(s)in accordance with the received intended state data. In someembodiments, the CP worker(s) 822 may provide service-specificorchestration operations. The CP worker(s) 822 may be communicativelycoupled to any suitable number of services (e.g., service(s) 812)including any suitable combination of a compute service, a storageservice, etc.). In some embodiments, the CP worker(s) 822 may beconfigured to provide instructions to data plane (DP) manager 824 forconfiguring one or more DP resources. DP manager 824 (e.g., an exampleof the hypervisor service 708 of FIG. 7 ) may be configured to create,modify, and/or remove or delete any suitable DP resource.

In some embodiments, DP manager 824 may be configured to manage anysuitable number of computing components (e.g., the DP resource(s) 116which may be, collectively, an example of a computing cluster). In someembodiments, the DP manager 824 may be configured to manage any suitabletype of computing cluster (e.g., a Kubernetes cluster, a set ofcomputing nodes used to execute containerized applications, etc.). TheCP worker(s) 822 may be configured to execute any suitable operations tocause the DP manager 824 to execute any suitable orchestration operationon the DP resource(s) 816 in accordance with the instructions identifiedby in-memory workflow manager 820 to configure the DP resource(s) 816 inaccordance with the intended state data. In some embodiments, CPmonitoring component 818 may be communicatively coupled to DPresource(s) 816 and configured to monitor the health of DP resource(s)816. In some embodiments, CP monitoring component 818 may be configuredto transmit (e.g., to a user device of user 114 using any suitable formof electronic communication) any suitable health data indicating thehealth of one or more of the DP resource(s) 816. By way of example, CPmonitoring component 818 may transmit any suitable health data viaControl plane API 806 to user device 810. As another example, userdevice 810 may be configured to request (e.g., via CP API 806) and/orobtain current state data and/or heath data from the CP data store 814periodically, or according to a predefined schedule.

In some embodiments, the CP monitoring component 818 may be configuredto monitor and assess current state data of the DP resource(s) 816. Insome embodiments, the CP monitoring component 818 may receive currentstate data store/update current state data of the DP resource(s) 816within CP data store 814. Current state data may be provided by DPmanager 824 to a corresponding CP worker, which in turn may provide thecurrent state data to the in-memory workflow manager 820, which may thenprovide the current state data to the CP monitoring component 818. Insome embodiments, CP worker(s) 822 and/or in-memory workflow manager 820may update CP data store 814 directly with the current state data of anysuitable DP resource such that the current state data of a given DPresource may be retrieved by the CP monitoring component 818 at anysuitable time.

Although CP monitoring component 818, in-memory workflow manager 820,and CP worker(s) 822 are depicted as separate components of controlplane 802, in some embodiments, any suitable combination of CPmonitoring component 818, in-memory workflow manager 820, and/or CPworker(s) 822 may be provided by one service (e.g., worker service 706of FIG. 7 , an example of one of the services of service(s) 502 of FIG.5 ).

FIG. 9 is a block diagram depicting a number of cloud-computingenvironments 900 provided by a computing cluster including two edgecomputing devices (e.g., edge devices 900A and 900B, each being anexample of the edge devices of FIGS. 1-6 ), according to at least oneembodiment. Each cloud-computing environment may include a control planeand a data plane. By way of example, edge device 900A may includecontrol plane 902A and data plane 904A. Edge device 900B may includecontrol plane 902B and data plane 904B. In some embodiments, controlplane 902A and 902B may operate as a distributed control plane and dataplane 904A and 904B may operate as a distributed data plane.

The user device 910 may be configured to transmit work requests to edgedevice 900A. In some embodiments, the edge device 900A may be the soleedge device to include a public network interface (e.g., NIC2 and/or NIC4 of FIG. 5 ) through which user device 910 may transmit work requestsand receive status on previously transmitted work requests. Thus, insome embodiments, edge device 900A alone may be configured to receivedata from a public network (e.g., network 520) through which user device910 and edge device 900A may be communicatively connected.

In some embodiments, the control plane 902A may be responsible foraccepting work requests that include intended state data that describesan intended state of a set of one or more data plane resources. Forexample, a work request may be received by control plane applicationprogramming interface (API) 906A. The work request can be initiated byuser 908 via user device 910 interfacing with control plane 902A. Insome embodiments, control plane API 906A (e.g., an example of the APIservice 702 of FIG. 7 ) may be one of the service(s) 912A (an example ofthe service(s) 502 of FIG. 5 ). Control plane API 906A may be configuredto receive any suitable number of work requests corresponding to one ormore data resources (e.g., a virtual machine, a cluster of virtualmachines, etc.) from the user device 910.

In some embodiments, the control plane API 906A may be configured tostore received work requests with CP data store 914A. In someembodiments, CP data store 914A and 914B (collectively referred to as“CP data store 914) form a distributed storage which is accessible tothe edge devices 900A and 900B. As described above, these work requestsmay include a request identifier and intended state data correspondingto an intended state of the data plane 904A and 904B, which may operateas a distributed data plane (referred to as “data plane 904). In someembodiments, the CP data store 915 may be configured to store a mappingof one or more data plane identifiers (DPIDs) of DP resource(s) 916A and916B with intended state data and/or current state data.

Each edge device may include a corresponding (CP) monitoring servicewithin its corresponding control plane. By way of example, the edgedevice 900A may include CP monitoring service 918A and edge device 900Bmay include CP monitoring service 918B. The CP monitoring services 918Aand 918B may individually be a service that is configured toperiodically (e.g., according to a predetermined frequency, schedule,etc.) determine whether the intended state data received by the controlplane API 906A and stored in the CP data store 915 (e.g., from apreviously received work request) matches current state data stored fora corresponding DP resource (if that DP resource currently exists). Eachof the CP monitoring services 918A and 918B may be communicativelycoupled to corresponding service(s) 912A and 912B (e.g., via substratenetwork 504 of FIG. 5 ), respectively. Service(s) 912A and 912B mayinclude any suitable number of cloud computing services configured tomanage billing, identity, authorization, and the like. In someembodiments, CP monitoring services 918A and/or 918B is one of theservice(s) 912A and/or 912B, respectively. In some embodiments, the CPmonitoring service of each edge device may be communicatively coupled toa corresponding in-memory workflow manager (e.g., in-memory workflowmanager 920A and 920B, respectively) and may be configured to invoke thefunctionality provided by the corresponding in-memory workflow manager.By way of example, if the CP monitoring service 918A or 918B determinesthat the current state data of a DP resource is not in line with (e.g.,does not match) the intended state data stored in CP data store 914,that CP monitoring service may invoke the functionality of thecorresponding in-memory workflow manager to rectify the discrepancy.

In some embodiments, in-memory workflow managers 920A and 920B may beindividually configured to identify one or more predefined workflowswhich individually identify operations to perform to configure DPresource(s) 916A and 916B, respectively, in accordance withcorresponding intended state data. In some embodiments, the in-memoryworkflow manager 920A may be configured to initiate one or more workersof control plane (CP) worker(s) 922A and forward the workflowinstructions and/or intended state data to a given CP worker to performthe operations related to configuring the corresponding DP resource(s)916A in accordance with the received intended state data. Similarly, thein-memory workflow manager 920B may be configured to initiate one ormore workers of control plane (CP) worker(s) 922B and forward theworkflow instructions and/or intended state data to a given CP worker toperform the operations related to configuring the corresponding DPresource(s) 916B in accordance with the received intended state data. Insome embodiments, the CP worker(s) 922A and/or 922B may provideservice-specific orchestration operations. The CP worker(s) 922A and/or922B may be communicatively coupled to any suitable number of services(e.g., service(s) 912A and 912B, respectively) including any suitablecombination of a compute service, a storage service, etc.). In someembodiments, the CP worker(s) 922A and/or 922B may be configured toprovide instructions to data plane (DP) manager 924A or DP manager 924B,respectively, for configuring one or more DP resources. DP manager 924Aand/or 924B may each be an example of the hypervisor service 708 of FIG.7 and may each be configured to create, modify, and/or remove or deleteany suitable DP resource.

In some embodiments, DP manager 924A and 924B may be configured tomanage any suitable number of computing components (e.g., the DPresource(s) 916A and 916B, respectively which may be, collectively, anexample of a computing cluster). In some embodiments, the DP manager924A and/or 924B may be configured to manage any suitable type ofcomputing cluster (e.g., a Kubernetes cluster, a set of computing nodesused to execute containerized applications, etc.). The CP worker(s) 922Aand/or 922B may be configured to execute any suitable operations tocause the corresponding DP manager to execute any suitable orchestrationoperation on the corresponding DP resource(s) in accordance with theinstructions identified by the corresponding in-memory workflow managerto configure the corresponding DP resource(s) in accordance with theintended state data. In some embodiments, CP monitoring services 918Aand 918B may be communicatively coupled to corresponding DP resource(s)916A and 916B, respectively, and may be configured to monitor the healthof those corresponding DP resource(s). In some embodiments, CPmonitoring service 918A and/or 918B may be configured to transmit (e.g.,to a user device of user 908 using any suitable form of electroniccommunication) any suitable health data indicating the health of one ormore of the DP resource(s) 916A and/or 916B. By way of example, CPmonitoring service 918A may transmit any suitable health data viaControl Plane API 906A to user device 910. As another example, userdevice 810 may be configured to request (e.g., via CP API 906A) and/orobtain current state data and/or heath data from the CP data store 915periodically, or according to a predefined schedule.

The CP workers may be configured to instruct any suitable DP manager,regardless of whether the DP manager resides at the same edge device. Byway of example, CP worker(s) 922A may be configured to instruct DPmanager 924B to create, modify, and/or delete/remove any suitable numberof DP resource(s) 916B. Likewise, CP worker(s) 922B may be configured toinstruct DP manager 924A to create, modify, and/or delete/remove anysuitable number of DP resource(s) 916A. In some embodiments, CPmonitoring service 918A may be configured to store health data within CPdata store 914A and CP monitoring service 918B may be configured tostore health data within CP data store 914B. In some embodiments,control plane API 906A may poll for current status of DP resource(s)916A and/or 916B (collectively, referred to as “DP resource(s) 916) fromCP data store 914. Thus, the health data and/or current state data ofany given DP resource may be obtained (e.g., using the control plane API906A) by the user device 910 via the public network interface betweenedge device 900A and user device 910. In some embodiments, the userdevice 910 may poll for/request current state data periodically (e.g.,every 1-2 seconds after submitting a work request).

In some embodiments, the CP monitoring services 918A and 918B may eachbe configured to monitor and assess current state data of the DPresource(s) 916A and 916B, respectively. In some embodiments, each CPmonitoring service may receive current state data store/update currentstate data of the corresponding DP resource(s) within CP data store 814.Current state data may be provided by DP manager 924A or 924B to acorresponding CP worker, which in turn may provide the current statedata to the in-memory workflow manager 920A or 920B, which may thenprovide the current state data to the CP monitoring service 918A or918B. In some embodiments, CP worker(s) 922A and/or 922B and/orin-memory workflow manager 920A and/or 920B may update CP data store 814directly with the current state data of any suitable DP resource suchthat the current state data of a given DP resource may be retrieved bythe CP monitoring service 918A or 918B at any suitable time.

FIG. 10 illustrates an example flow 1000 for performing one or moreoperations in connection with a work request, in accordance with atleast one embodiment. The flow 1000 may be performed by control plane(CP) monitoring service 1002 (e.g., an example of the CP monitoringservices 918A and/or 918B of FIG. 9 , CP monitoring component 818 ofFIG. 8 , etc.), in-memory workflow manager 1004 (e.g., an example of thein-memory workflow manager 920A and/or 920B of FIG. 9 , in-memoryworkflow manager 820 of FIG. 8 , etc.), and in-memory worker 1006 (e.g.,an example of the CP worker(s) 922A and/or 922B of FIG. 9 , CP worker(s)822 of FIG. 8 , etc.). In some embodiments, any suitable combination ofthe CP monitoring service 1002, the in-memory workflow manager 1004, andthe in-memory worker 1006 may be provided by a common service (e.g., oneof the service(s) 502 of FIG. 5 ).

The flow 1000 may begin at 1008, where the CP monitoring service 1002may invoke (e.g., via function call) the functionality of in-memoryworkflow manager 1004. The call performed at 1008 may include a requestidentifier for the work request. The call performed at 1008 may beexecuted in response to the CP monitoring service 1002 identifying thatthe current state data for a given DP resource does not match theintended state data for that resource.

At 1010, in-memory workflow manager 1004 my determine, from a portion ofa data store (e.g., a portion of the CP data store 915 of FIG. 9 , aportion of CP data store 814 of FIG. 8 , etc.) whether another componenthas begun executing operations corresponding to the work request. By wayof example, the in-memory workflow manager 1004 may check within aportion of the data store that is dedicated to storing requestidentifiers corresponding to work requests that a control plane workerhas commenced processing. If the in-memory workflow manager 1004 checksthe dedicated portion of the data store and the request identifier hasalready been stored in that dedicated portion of the data store, thenthe in-memory workflow manager 1004 may determine that the request isalready being (or has already been) handled by another computingcomponent (e.g., another in-memory workflow manager of another edgedevice) and in-memory workflow manager 1004 may send a response at 1012to CP monitoring service 1002 that indicates the request is beinghandled (or has been handled) by another computing component.Alternatively, the in-memory workflow manager 1004 may determine thatthe request identifier has not been stored in the dedicated portion ofthe data store. In this scenario, the in-memory workflow manager 1004may determine that the request has not yet been handled. If so, the flow1000 may proceed to 1014.

At 1014, the in-memory workflow manager 1004 may create an entry for thework request within the dedicated portion of the data store. In someembodiments, the request identifier may be stored within the dedicatedportion of the data store to indicate that a computing component (e.g.,in-memory workflow manager 1004) has commenced processing the request.In some embodiments, the dedicated portion of the data store (e.g., adistributed data store such as the CP data store 915 of FIG. 9 , the CPdata store 814 of FIG. 8 , etc.) may be accessible to any suitablein-memory workflow manager operating at any suitable edge device of acomputing cluster. By storing the request identifier within thededicated portion of the data store, in-memory workflow manager 1004ensures that it is the only computing component to process the workrequest. In this manner, in-memory workflow manager 1004 can ensure thatduplicate processing of the request by another computing component isavoided.

At 1016, the in-memory workflow manager 1004 may identify one or morepredefined workflows for configuring one or more data plane resource(e.g., one or more virtual machines) to be in accordance with theintended state data corresponding to the request identifier (asretrieved from the data store).

At 1018, the in-memory workflow manager 1004 may instantiate in-memoryworker 1006 (e.g., an example of an execution thread). In-memoryworkflow manager 1004 may provide the one or more workflows to thein-memory worker 1006 for execution.

At 1020, in-memory worker 1006 may execute any suitable operationscorresponding to the one or more workflows identified by the in-memoryworkflow manager 1004.

At 1022, and once the operations performed at 1020 are complete, thein-memory worker 1006 may provide status of the workflow to in-memoryworkflow manager 1004. In some embodiments, the status may includecurrent state data indicating parameters which specify a current stateof the DP resource(s) affected by the work request.

At 1024, the in-memory workflow manager 1004 may forward the status ofthe DP resource(s) to the CP monitoring service 1002. Alternatively, thein-memory worker 1006 and/or the in-memory workflow manager 1004 maystore the status and/or current state data within a data storeaccessible to the CP monitoring service 1002, such as the CP data stores814 and/or 914 of FIGS. 8 and 9 , respectively.

FIG. 11 is a block diagram illustrating an example method 1100 forproviding in-memory workflow management at an edge computing device, inaccordance with at least one embodiment. The method 1100 may beperformed by any suitable number of edge devices (e.g., the edge devices900A and 900B of FIG. 9 ). In some embodiments, the method 1100 mayinclude more or fewer steps than the number depicted in FIG. 11 . Itshould be appreciated that the steps of method 1100 may be performed inany suitable order.

The method 1100 may begin at 1102, where a plurality of respectivehosting environments (e.g., data planes 904A and 904B of FIG. 9 ) may beimplemented within a computing cluster comprising a plurality of edgecomputing devices (e.g., the edge devices 900A and 900B). In someembodiments, the plurality of edge computing devices implement adistributed control plane (e.g., collectively, control planes 902A and902B) for managing respective data planes (e.g., data planes 904A and904B) of the plurality of edge computing devices.

At 1104, a work request may be obtained by an in-memory workflow managerof the distributed control plane. By way of example, in-memory workflowmanager 920B may obtain a work request. In some embodiments, the workrequest may comprise a request identifier and intended state datacorresponding to a respective data plane of a particular cloud-computingedge device of the plurality of edge computing devices. For example, thework request may include intended state data corresponding to a DPresource(s) 916B of data plane 904B of FIG. 9 .

At 1106, using the request identifier, it may be determined that thework request has yet to be started. By way of example, the in-memoryworkflow manager 920B may identify that the request identifier has notyet been stored in a dedicated portion of CP data store 915 of FIG. 9 .As a result, the in-memory workflow manager 920B may determine that thework request has not yet been processed by a computing component (e.g.,the in-memory workflow manager 920B itself, or any suitable computingcomponent of the edge device cluster comprising edge devices 900A and900B).

At 1108, the in-memory workflow manager 920B may initialize an in-memoryexecution thread (e.g., one of CP worker(s) 922B of FIG. 9 ) to executeone or more orchestration tasks associated with configuring therespective data plane according to the intended state data. By way ofexample, the in-memory workflow manager 920B may instantiate thein-memory execution thread (e.g., one of the CP worker(s) 922B of FIG. 9) and pass one or more instructions that indicate the in-memoryexecution thread is to execute the one or more orchestration tasks. Thein-memory execution thread may be configured to instruct a DP manager(e.g., DP manager 924B of FIG. 9 , an example of the hypervisor service708 of FIG. 7 ) to perform one or more operations to configure one ormore of the DP resource(s) 916B in accordance with the intended statedata.

At 1110, current state data indicating a current state of the respectivedata plane of the particular cloud-computing edge device may bereceived. The current state data may be stored (e.g., in CP data store915).

At 1112, an indication of the current state of the respective data planemay be provided (e.g., to the user device 910). By way of example, insome embodiments, the current state data may be retrieved (e.g., by theuser device 910 using the control plane API 906A) and provided to theuser device 910 of FIG. 9 . Subsequently, the user device 910 maypresent the current state data.

Although the example above includes modifying DP resource(s) of an edgedevice by an in-memory workflow manager operating at that edge device,embodiments include using an in-memory workflow manager and/or CPworker(s) to instruct a DP manager of another edge device to modify theDP resources of that other edge device. Thus, any in-memory workflowmanager of any suitable edge device may be utilized to performmodifications to any suitable data plane of any suitable edge device inan edge device cluster.

Although specific embodiments have been described, variousmodifications, alterations, alternative constructions, and equivalentsare also encompassed within the scope of the disclosure. Embodiments arenot restricted to operation within certain specific data processingenvironments, but are free to operate within a plurality of dataprocessing environments. Additionally, although embodiments have beendescribed using a particular series of transactions and steps, it shouldbe apparent to those skilled in the art that the scope of the presentdisclosure is not limited to the described series of transactions andsteps. Various features and aspects of the above-described embodimentsmay be used individually or jointly.

Further, while embodiments have been described using a particularcombination of hardware and software, it should be recognized that othercombinations of hardware and software are also within the scope of thepresent disclosure. Embodiments may be implemented only in hardware, oronly in software, or using combinations thereof. The various processesdescribed herein can be implemented on the same processor or differentprocessors in any combination. Accordingly, where components or modulesare described as being configured to perform certain operations, suchconfiguration can be accomplished, e.g., by designing electroniccircuits to perform the operation, by programming programmableelectronic circuits (such as microprocessors) to perform the operation,or any combination thereof. Processes can communicate using a variety oftechniques including but not limited to conventional techniques forinter process communication, and different pairs of processes may usedifferent techniques, or the same pair of processes may use differenttechniques at different times.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that additions, subtractions, deletions, and other modificationsand changes may be made thereunto without departing from the broaderspirit and scope as set forth in the claims. Thus, although specificdisclosure embodiments have been described, these are not intended to belimiting. Various modifications and equivalents are within the scope ofthe following claims.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (i.e., meaning“including, but not limited to,”) unless otherwise noted. The term“connected” is to be construed as partly or wholly contained within,attached to, or joined together, even if there is something intervening.Recitation of ranges of values herein are merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range, unless otherwise indicated herein and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein, isintended merely to better illuminate embodiments and does not pose alimitation on the scope of the disclosure unless otherwise claimed. Nolanguage in the specification should be construed as indicating anynon-claimed element as essential to the practice of the disclosure.

Disjunctive language such as the phrase “at least one of X, Y, or Z,”unless specifically stated otherwise, is intended to be understoodwithin the context as used in general to present that an item, term,etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y,and/or Z). Thus, such disjunctive language is not generally intended to,and should not, imply that certain embodiments require at least one ofX, at least one of Y, or at least one of Z to each be present.

Preferred embodiments of this disclosure are described herein, includingthe best mode known for carrying out the disclosure. Variations of thosepreferred embodiments may become apparent to those of ordinary skill inthe art upon reading the foregoing description. Those of ordinary skillshould be able to employ such variations as appropriate and thedisclosure may be practiced otherwise than as specifically describedherein. Accordingly, this disclosure includes all modifications andequivalents of the subject matter recited in the claims appended heretoas permitted by applicable law. Moreover, any combination of theabove-described elements in all possible variations thereof isencompassed by the disclosure unless otherwise indicated herein.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

In the foregoing specification, aspects of the disclosure are describedwith reference to specific embodiments thereof, but those skilled in theart will recognize that the disclosure is not limited thereto. Variousfeatures and aspects of the above-described disclosure may be usedindividually or jointly. Further, embodiments can be utilized in anynumber of environments and applications beyond those described hereinwithout departing from the broader spirit and scope of thespecification. The specification and drawings are, accordingly, to beregarded as illustrative rather than restrictive.

What is claimed is: 1-20. (canceled)
 21. A computer-implemented method,comprising: implementing, by an edge computing device of a plurality ofedge computing devices of a computing cluster, a hosting environmentcomprising at least part of a distributed control plane implemented bythe plurality of edge computing devices, the distributed control planebeing configured to manage respective data planes of the plurality ofedge computing devices; obtaining, by a computing component of thedistributed control plane executed by the edge computing device, a workrequest comprising intended state data corresponding to a respectivedata plane of a particular cloud-computing edge device of the pluralityof edge computing devices; executing, by the computing component,operations to cause one or more orchestration tasks to be executed, theone or more orchestration tasks being associated with configuring therespective data plane according to the intended state data; obtaining,by the computing component, current state data indicating a currentstate of the respective data plane of the particular cloud-computingedge device; and providing, by the computing component, an indication ofthe current state of the respective data plane.
 22. Thecomputer-implemented method of claim 1, wherein the work request isinitiated by from a user device configured to communicate with one edgecomputing device of the plurality of edge computing devices.
 23. Thecomputer-implemented method of claim 1, wherein the computing componentis an in-memory workflow manager provided by a container executing onthe edge computing device.
 24. The computer-implemented method of claim1, wherein the one or more orchestration tasks associated withconfiguring the respective data plane according to the intended statedata are executed in response to: obtaining, from a distributed datastore accessible to the plurality of edge computing devices, historicalstate data indicating respective current states of the respective dataplane of the particular cloud-computing edge device; comparing thehistorical state data and the intended state data; and determining thatthere is a difference between the historical state data and the intendedstate data obtained from the distributed data store.
 25. Thecomputer-implemented method of claim 1, wherein the work request furthercomprises a request identifier, and wherein the computer-implementedmethod further comprises determining, by the computing component thatthe work request has yet to be started based at least in part ondetermining that a record associated with the request identifier is notyet stored in a distributed data store accessible to the plurality ofedge computing devices.
 26. The computer-implemented method of claim 1,wherein the plurality of edge computing devices are communicativelycoupled with one another via a substrate network that is different froma public network.
 27. The computer-implemented method of claim 1,wherein the computing component executing on the edge computing deviceidentifies the one or more orchestration tasks associated withconfiguring the respective data plane according to the intended statedata.
 28. An edge computing device operating as part of a computingcluster of a plurality of edge computing devices, the edge computingdevice comprising: one or more processors; and one or more memoriesconfigured with computer-executable instructions that, when executed bythe one or more processors, cause the edge computing device to:implement a hosting environment within the computing cluster, thehosting environment comprising at least part of a distributed controlplane implemented by the plurality of edge computing devices, thedistributed control plane being configured to manage respective dataplanes of the plurality of edge computing devices; obtain, by acomputing component of a portion of the distributed control planeexecuted by the edge computing device, a work request comprisingintended state data corresponding to a respective data plane of the edgecomputing device; initialize, by the computing component, operations tocause one or more orchestration tasks to be executed, the one or moreorchestration tasks being associated with configuring the respectivedata plane according to the intended state data; receive current statedata indicating a current state of the respective data plane of the edgecomputing device; and provide an indication of the current state of therespective data plane.
 29. The edge computing device of claim 8, whereinthe work request is initiated by a user device configured to communicatewith one edge computing device of the plurality of edge computingdevices.
 30. The edge computing device of claim 8, wherein the computingcomponent is an in-memory workflow manager provided by a containerexecuting on the edge computing device.
 31. The edge computing device ofclaim 8, wherein the one or more orchestration tasks associated withconfiguring the respective data plane according to the intended statedata are executed in response to: obtaining, from a distributed datastore accessible to the plurality of edge computing devices, historicalstate data indicating respective current states of the respective dataplane of the particular cloud-computing edge device; comparing thehistorical state data and the intended state data; and determining thatthere is a difference between the historical state data and the intendedstate data obtained from the distributed data store.
 32. The edgecomputing device of claim 8, wherein the work request further comprisesa request identifier, and wherein executing the computer-executableinstructions further causes the edge computing device to determine thatthe work request has yet to be started based at least in part ondetermining that a record associated with the request identifier is notyet stored in a distributed data store accessible to the plurality ofedge computing devices.
 33. The edge computing device of claim 8,wherein the plurality of edge computing devices are communicativelycoupled with one another via a substrate network that is different froma public network.
 34. The edge computing device of claim 8, wherein thecomputing component executing on the edge computing device identifiesthe one or more orchestration tasks associated with configuring therespective data plane according to the intended state data.
 35. Anon-transitory computer-readable storage medium comprisingcomputer-executable instructions that, when executed with one or moreprocessors of an edge computing device, cause the edge computing deviceto: implement a hosting environment within the computing cluster, thehosting environment comprising at least part of a distributed controlplane implemented by the plurality of edge computing devices, thedistributed control plane being configured to manage respective dataplanes of the plurality of edge computing devices; obtain, by acomputing component of a portion of the distributed control planeexecuted by the edge computing device, a work request comprisingintended state data corresponding to a respective data plane of the edgecomputing device; initialize, by the computing component, operations tocause one or more orchestration tasks to be executed, the one or moreorchestration tasks being associated with configuring the respectivedata plane according to the intended state data; receive current statedata indicating a current state of the respective data plane of the edgecomputing device; and provide an indication of the current state of therespective data plane.
 36. The non-transitory computer-readable storagemedium of claim 15, wherein the computing component is an in-memoryworkflow manager provided by a container executing on the edge computingdevice.
 37. The non-transitory computer-readable storage medium of claim15, wherein the one or more orchestration tasks associated withconfiguring the respective data plane according to the intended statedata are executed in response to: obtaining, from a distributed datastore accessible to the plurality of edge computing devices, historicalstate data indicating respective current states of the respective dataplane of the particular cloud-computing edge device; comparing thehistorical state data and the intended state data; and determining thatthere is a difference between the historical state data and the intendedstate data obtained from the distributed data store.
 38. Thenon-transitory computer-readable storage medium of claim 15, wherein thework request further comprises a request identifier, and wherein thecomputer-implemented method further comprises determining, by thecomputing component that the work request has yet to be started based atleast in part on determining that a record associated with the requestidentifier is not yet stored in a distributed data store accessible tothe plurality of edge computing devices.
 39. The non-transitorycomputer-readable storage medium of claim 15, wherein the plurality ofedge computing devices are communicatively coupled with one another viaa substrate network that is different from a public network.
 40. Thenon-transitory computer-readable storage medium of claim 15, wherein thecomputing component executing on the edge computing device identifiesthe one or more orchestration tasks associated with configuring therespective data plane according to the intended state data.